Skip to content

Test which doses differ from the vehicle with multiple contrast tests - #82

Merged
parashardhapola merged 16 commits into
masterfrom
dose/04-multivariate-t-and-contrast-tests
Nov 20, 2026
Merged

parashardhapola merged 16 commits into
masterfrom
dose/04-multivariate-t-and-contrast-tests

Conversation

@parashardhapola

Copy link
Copy Markdown
Member

The problem

A dose series asks, for each gene, which doses differ from the vehicle, with one error rate for the gene's comparisons. In a survey of 130 published single-cell studies with three or more dose levels, 76 analysed dose as separate contrasts against a control, and 70 profiled exactly three dose levels, usually a control and two doses. Clinical dose finding asks the same question: of 96 phase 2 dose-ranging trials in NEJM, The Lancet and JAMA (2018 to 2026) whose primary analysis could be determined, 39 used pairwise tests with multiplicity control, 27 unadjusted pairwise tests and 15 MCP-Mod.

Dunnett's test (each dose against the control; Dunnett 1955), the Williams-type test (the vehicle against the size-weighted mean of the highest one, two or more doses; Bretz 2006) and the MCP step of MCP-Mod refer the largest of correlated t-statistics to the multivariate t distribution (Hothorn, Bretz and Westfall 2008), which CytePool could not compute. Bonferroni's correction ignores the correlation that the shared vehicle creates, 0.5 between two doses of groups the size of the vehicle's: at single-step p-values from 1e-2 to 1e-6, Bonferroni's p-values are 22% to 26% larger than the single-step ones for three such contrasts at 5 degrees of freedom, and 65% to 73% larger for eight. The gap falls as the degrees of freedom grow and the p-values shrink, to at most 7% and 15% for z-tests (Dunnett's integral). An F-test of the contrasts, which PR 2 added, says whether any dose differs, not which.

The p-values also have to be accurate where calls are made. At a false discovery rate of 5%, the Benjamini-Hochberg cut-off for the top gene of 20,000 is 2.5e-6, while mvtnorm, which multcomp and DoseFinding use, defaults to an absolute error of 1e-3 (GenzBretz(maxpts = 25000, abseps = 0.001); Genz and Bretz 2002). Asked for an absolute error of 1e-12 on three correlated t-statistics with 10 degrees of freedom, mvtnorm 1.4-2 returned 4.03e-6 for a true 4.37e-6, 8% low and twice its own stated error, and 0 for a true 5.9e-9, with a stated error of 0; in a probe with another seed it returned 2.58e-6, 41% low and 16 times its stated error.

The solution

Two modules. cytepool.stats.multiple_contrasts builds Dunnett's and Williams-type contrasts and runs single-step multiple contrast tests over every gene, from a moderated fit, or from estimates and their covariances as multcomp's glht takes them with parm():

from cytepool import Design, limma_voom, simulate_dose_series
from cytepool.stats.multiple_contrasts import dunnett_contrasts, multiple_contrast_test

cube, truth = simulate_dose_series([0, 1, 3, 10, 30], replicates=3, n_genes=2000, rng=3)
design = Design(cube.samples, "~ 0 + dose_group")
result = limma_voom(cube, design, contrasts=dunnett_contrasts(design.columns))
test = multiple_contrast_test(result.fit("cell_type_1").moderated, rng=1)
test.table()                               # each gene's max-t test, Benjamini-Hochberg over genes
test.table("dose_groupd4 - dose_groupd0")  # one contrast, adjusted for the gene's contrasts
  • The tests. For each gene: the contrasts' estimates, standard errors and t-statistics, and single-step adjusted p-values P(max_l |T_l| >= |t_j|), or their one-sided forms, from the multivariate t on the gene's degrees of freedom with the correlation of C' V C. A moderated fit gives its coefficients, their covariance (the posterior variance times the fit's unscaled covariance, per gene or common to all genes) and df_total; a fit with limma's approximate standard errors of contrasts holds no covariance of its contrasts and is refused. Degrees of freedom need not be integers, as moderated ones are not. table() gives each gene's test as a whole: its most extreme contrast, that contrast's estimate and p-value, and Benjamini and Hochberg's adjustment over the genes. table(contrast) gives that contrast's p-value adjusted for the gene's contrasts, which the documentation reads at stageR's confirmation level, alpha R / G, after the screen of the genes (Van den Berge et al. 2017). Within a gene, p-values never decrease as t becomes less extreme. The result keeps the correlation of each gene's contrasts, from which multivariate_t_quantile gives simultaneous confidence intervals; for group means they agree with scipy's dunnett within scipy's own Monte Carlo error.
  • The contrasts. dunnett_contrasts(groups, control=...) names each contrast "dose - vehicle", as multcomp does. williams_contrasts(sizes) takes the size of each named group, weighs the j highest doses by their sizes and names each contrast by the doses it pools, such as "mid to high - vehicle", where multcomp writes "C 2"; tested two-sided for genes, whose direction is not known, Williams-type contrasts test rises and falls at one error rate. Both builders return a Contrasts, matched to the estimates by name, and both refuse a model's intercept among the groups, and sets, which have no order.
  • The probabilities. cytepool.stats.multivariate_t gives multivariate_t_cdf, the probability of a box and its complement, and multivariate_t_quantile, the critical value of a max-t test, each with its standard error. The complement is a sum over the coordinates of the chance that each is the first to exceed its limit; each term is computed by conditioning on its exceedance, with randomly shifted lattice points (Genz 1992; Genz and Bretz 2002), which keeps the relative error bounded however small the p-value (Owen, Maximov and Chertkov 2019). Each p-value carries a standard error of at most 1% of itself by default (relative_error). Two contrasts, as a vehicle and two doses give, are computed by a deterministic quadrature to 1e-12 relative, without random numbers.

"Multiple testing" gains a section, "Several contrasts of one gene", with this example, the Bonferroni gaps by degrees of freedom and simultaneous intervals; "Choosing a method" and the glossary gain entries. The bulk RNA-seq guide tests both drugs against the control of its simulated experiment at one error rate per gene: the max-t test calls 483 genes, 5.4% of them unchanged, against 508 and 4.5% for the F-test, and stageR's rule then names a drug in each called gene, with 27 of the 483 genes (5.6%) carrying a false call, 26 of them genes that neither drug changed. The dose response guide, from PR 7, now prints the first five rows of its spline table rounded, F to 1 decimal and p-values to 2 significant digits, as the bulk guide prints its strongest genes: CI compares printed text exactly, and full-precision floats can differ in their last digits on runners with AVX-512.

Why it matters

PR 7 fits one moderated model of the dose groups in each cell type. PR 8 runs Dunnett's, Williams-type and MCP-Mod tests of that fit through multiple_contrast_test, takes MCTtest's critical value from multivariate_t_quantile, finds the lowest effective dose by a closed test, adjusts per-dose contrasts stage-wise after the gene-level screen, and adds optimal_contrasts to cytepool.stats.multiple_contrasts. Because each p-value keeps its relative accuracy down to the smallest doubles, the calls that Benjamini and Hochberg make across 20,000 genes change with the seed only for p-values within about 2% of their cut-off.

On 20,000 genes and 2 threads, in five runs on three kinds of Modal host, Dunnett's test of 8 doses took 0.60 to 0.88 ms per gene, 76 to 106 times the baseline (the weighted fit of 20,000 genes by 200 samples that linear_models.py --benchmark times), within the target of 1 ms per gene, about 120 baselines. Two contrasts took 0.02 to 0.04 ms per gene. The Williams-type test of 8 one-sided contrasts took 1.13 to 1.67 ms per gene (143 to 193 baselines), above the target on every host. In a probe on 2,000 of these genes, a Williams-type problem took 307 points at the mean against 223 for Dunnett's, and p-values of at least 0.5 took 70% of the Williams-type points (94% of Dunnett's), as the stopping rule holds the smaller of p and 1 - p to its relative error. A tenfold smaller relative_error took 12.0 to 12.6 times as long.

Validation

validation/multiple_contrast_tests.py ran on Modal twice in one container (4 CPU, 8 GiB, an Intel Granite Rapids host, with the settings that make numpy and OpenBLAS compute as without AVX-512) and printed the same 118 checks both times: 106 pass, 6 information, 3 deliberate differences, 3 controls, none failed. The second run took 7 min 45 s with a peak memory of 475 MiB, as the Validation page reports. Every Monte Carlo problem has its own seed, and CytePool runs at relative_error=1e-3 in the calibrated checks, whose tolerance is 5.5 combined standard errors: the 99.994% point of the t distribution on the 15 degrees of freedom of CytePool's errors, whose two-sided tail is that of 4 normal standard errors. mvtnorm's and multcomp's standard error is their error attribute over 3, as 200 repeated calls measured; scipy's comes from the spread of 16 calls and ALOE's from its own draws.

Check Reference Largest difference Tier
One coordinate, df 4.5, 28 limits from -30 to 1e8 mpmath 1.3.0, 30 digits 2.1e-14 exact, 1e-10
Two coordinates: 336 boxes, df 1 to 200 and the normal, correlations -0.95 to 0.99, complements to 1e-14, small boxes and tail orthants to 1.5e-87 Ding's conditional distribution, mpmath, 30 digits 1.3e-13; actual error at most 0.21 of the stated bound close, 1e-6
Product correlations (Dunnett's contrasts of unequal groups): k = 3 to 10, 4 df, 180 boxes, smaller side down to 4.0e-13 Dunnett's integral 3.82 standard errors calibrated
Product correlations with a limit per coordinate: k = 3, 5 and 8, 4 df, 96 boxes, tails split unevenly and small boxes, smaller side down to 5.8e-10 Dunnett's integral 3.19 standard errors calibrated
Rank-two correlations (Tukey's of 3 groups, Williams' of 3, random directions): 90 boxes, smaller side down to 2.5e-11 average over directions, mpmath, 30 digits 2.85 standard errors calibrated
Honest errors: root-mean-square of (estimate - reference) / error, 366 problems at relative_error 1e-3 and 369 at 0.01 with fresh seeds sqrt(15/13) = 1.07 1.115 and 1.107 within 4 simulation standard errors
Correlations of rank 3 and more without a product structure (Williams-type contrasts of 5, 7 and 9 groups, Tukey's of 4, random), df 3, 8 and the normal, 54 complements from 1e-4 to 1e-10 ALOE (Owen, Maximov and Chertkov 2019), 1e6 draws 2.75 standard errors calibrated
General correlations of 4 to 10 contrasts (Dunnett's, Williams', Tukey's singular, random), df 3 to 30 and the normal, 100 boxes from 0.5 to 1e-3 mvtnorm 1.4-2 pmvt 4.23 standard errors calibrated
The same at df 4.5, 10 and 20.5, 60 boxes scipy 1.18.1 multivariate_t.cdf 3.58 standard errors calibrated
Quantiles of two coordinates (36) roots of Ding's formula, mpmath 9.7e-16 close, 1e-6
Quantiles of product correlations (36), of general ones (40) and the round trip (76) roots of Dunnett's integral; pmvt at CytePool's quantile; multivariate_t_cdf 4.14, 2.67 and 3.12 standard errors calibrated
dunnett_contrasts (23 cases, names too) and williams_contrasts (14 cases), 2 to 9 groups multcomp 1.4-32 contrMat 0 identical; exact
multiple_contrast_test: estimates, standard errors and t of 160 features, 4 families, every alternative, df 5 to 40 and the normal multcomp glht(parm(b, V, df)) 4.1e-15 exact, 1e-10
multiple_contrast_test: 675 single-step p-values from 1e-3 to 1, of 760 multcomp summary(glht) 5.48 standard errors, multcomp's outlier (below) calibrated
Bonferroni's p-values from 1e-3 to 0.5, Dunnett's, Williams' and Tukey's families: the smallest gap multcomp adjusted("bonferroni") 43.2, 1994.6 and 60.5 standard errors control: above 5.5

Information rows: asked for an absolute error of 1e-12 with 1e7 points, mvtnorm's pmvt returned 4.025e-6 for a true 4.367e-6 (8% low, twice its stated error of 1.7e-7) and 0 for a true 5.9e-9; qmvt, whose root search keeps pmvt's default absolute error of 1e-3, differs from CytePool's 40 quantiles by at most 5.5e-3 relative; scipy's complements for the singular correlation of Tukey's contrasts of 4 groups reach 11.75 times Bonferroni's upper bound, which no probability can exceed, where CytePool agrees with mvtnorm; below 1e-3, which its default absolute error cannot resolve, multcomp's 85 p-values ranged from 216 times smaller to 4 times larger than CytePool's, and were 0 for 10 of them. The largest calibrated difference, 5.48 standard errors at a p-value of 0.0115 of random contrasts with a nearly singular correlation, is multcomp's: a probe on Modal found its value at the script's seed, 0.0097, 6.1 of its own standard deviations below the mean of 30 calls with other seeds (0.0114), which CytePool's 0.01146 matches (0.01147 at relative_error 1e-5). Deliberate differences: non-integer df, which pmvt refuses; Williams-type names that say which doses are pooled; Williams with two groups, which contrMat refuses.

Modal runs, each an ephemeral app in the image that holds the r-dose and python environments, 16 in all and about 3.6 container-hours:

What Why Resources and duration Result
Three probes of mvtnorm and multcomp Pin their behaviour as black boxes: contrMat, pmvt with non-integer, zero and infinite df, the meaning of its error attribute, its tail, qmvt and glht(parm()) 2 CPU, 4 GiB; 9 s, 11 s and 53 s The first two stopped at calls that R refused (contrMat with two groups, non-integer df); the third gave the numbers the design used
The script twice and --benchmark, in one container The first complete version 4 CPU, 8 GiB; 1,784 s Failed: multcomp's summary(glht) stopped on a feature with a very large t; the seeded effects were moderated and each multcomp call guarded
The same After that fix 4 CPU, 8 GiB; 2,394 s Failed at a key error in the script's comparison of quantiles, fixed
The same, with --write-fixtures After fixing mpmath's absolute tolerance in the references 4 CPU, 8 GiB; 2,562 s 108 checks passed in both runs; the fixtures this PR commits
The same The build's final code: multcomp compared where it resolves its p-values; two branches that could not run removed 4 CPU, 8 GiB; 1,606 s 109 checks passed in both runs; fixtures byte-identical
multcomp on the 14 two-sided features of the random contrasts, at the script's seeds and 30 others Where the 5.48 standard errors come from 2 CPU, 4 GiB; 133 s multcomp's outlier, above
The script twice with --write-fixtures, and --benchmark The code after review, with the rows of limits per coordinate and of ALOE 4 CPU, 8 GiB; 2,749 s, AMD Zen 3 118 checks passed in both runs; the earlier rows unchanged; fixtures byte-identical
The same The final code, after two error messages changed (the recorded digest counts the strings the code uses) 4 CPU, 8 GiB; 1,716 s, Intel Granite Rapids The numbers above, the same 118 checks; fixtures byte-identical
The documentation build of CI: Node 24, the API reference with warnings as errors, myst build --strict This machine has no Node.js 4 CPU, 8 GiB; 37 s Passed with no warnings, on the final commit
CI's docs job after rebasing onto #81: the site build and check-outputs of the bulk RNA-seq and dose response guides The rebased branch, on the documentation this machine cannot build 4 CPU, 8 GiB; 204 s, a host without AVX-512 Passed; both guides' outputs current
CI's docs job on a host with AVX-512, without the settings that make numpy and OpenBLAS compute as on AVX2: the site build, check-outputs of the bulk RNA-seq and dose response guides, and the same check of both guides before their tables were rounded CI compares printed text exactly, and runners with AVX-512 compute float64 logarithms, powers and exponentials with other last bits 4 CPU, 8 GiB; 4 s, AMD Zen 3 Stopped at once: the host has no AVX-512
The same, second try The same 4 CPU, 8 GiB; 5 s, AMD Zen 3 Stopped: no AVX-512
The same, third try The same 4 CPU, 8 GiB; 4 s, AMD Zen 3 Stopped: no AVX-512
The same, fourth try, after a 10-minute pause The same 4 CPU, 8 GiB; 3 s, AMD Zen 3 Stopped: no AVX-512

docs/changed_since_run.py lists 34 other scripts whose checked code changed after their recorded runs. This PR rewords the error messages of two shared helpers, contrast_matrix (which now names "the estimates" where it named the fit's coefficients) and coefficient_position, which 32 scripts reach; the wording cannot change their results, but the recorded digest counts the strings the code uses, so the digests of all 32 change. Three of them, dose_response_calibration.py, dose_response_tests.py and plotting.py, were current on master; the other 29 were listed on master already, because master changed their code after their runs, as were multiple_testing.py and simulate_dose_series.py, which this PR does not reach. The 34: activity.py, combined_p_values_and_replication.py, composition_tests.py, cube_collection.py, deseq2_tests.py, design.py, differential_expression.py, differential_expression_options.py, dose_axes.py, dose_design.py, dose_response_calibration.py, dose_response_tests.py, dream.py, empirical_bayes.py, gene_set_files_and_ora.py, gene_set_tests.py, gsea.py, interferon_stimulated_pbmcs.py, kenward_roger_and_f_tests.py, linear_models.py, meta_analysis.py, mixed_models.py, multiple_testing.py, negative_binomial_fits.py, pathway_scores.py, pca_and_dot_plots.py, plotting.py, quasi_likelihood.py, rotation_gene_set_tests.py, sample_quality_weights.py, simulate_dose_series.py, sumrank.py, variance_partitioning.py and voom.py.

Tests

tests/test_multivariate_t.py and tests/test_multiple_contrasts.py hold 131 tests, which run in 17 s once numba has cached the compiled routines; the eight fixture files of tests/data/multiple_contrast_tests/ (74 kB) come from the validation script's Modal run and hold only the references' values.

  • Against independent references. One coordinate against the closed forms of the t distribution at df 1 and 2 and the normal, and mpmath at df 4.5; 336 two-coordinate boxes against Ding's formula in mpmath at 30 digits, to 1e-9 relative and within the stored bound; equicorrelated normal boxes, limits that differ between coordinates (tails and small boxes, at df 6.5 and the normal) and small boxes at df 4 against Dunnett's integral computed in the test; 183 product-correlation boxes into the tail (complements to 1e-12, small boxes) and 90 rank-two boxes against the fixtures' Dunnett integral and average over directions; a coordinate fixed by the conditioning one against its closed form; general correlations against mvtnorm; quantiles against mpmath's and Dunnett's roots and by the round trip, and their errors over 100 seeds against the root of Dunnett's integral; contrast matrices against multcomp's contrMat, and Williams' weights by hand; multiple_contrast_test against multcomp's estimates, standard errors and t (1e-10) and p-values (5.5 combined standard errors, where multcomp resolves them), and one contrast against limma's moderated t-test (1e-12); a moderated fit's contrasts, weighted and not, against contrasts_fit and moderated_statistics; and the correlation of Dunnett's contrasts against lambda_i lambda_j by hand, with simultaneous intervals and p-values against scipy's dunnett.
  • Honest errors. The root-mean-square of (estimate - reference) / error over 200 seeds of two problems, and over 60 seeds deep in the normal tail and 60 of nearly collinear coordinates, lies within 4 simulation standard errors of sqrt(15/13); a doubled or a halved error falls outside.
  • Invariants. Bonferroni's bound (with scipy's t tails) and the product bound; one-sided <= two-sided <= 2 x one-sided; the sign and order of the coordinates; repeated coordinates count once; results independent of the order of the problems, of batching and of threads, bit for bit; "less" at t equals "greater" at -t exactly; p-values never decrease as t becomes less extreme; a two-sided limit of 0 gives a complement of exactly 1.
  • Edges and arguments. Infinite limits dropped from a general correlation, NaN in any argument, underflow deep in the tail and of conditioning draws, small boxes, rays parallel to a face, the point limit, singular correlations with rounding noise, a zero-variance contrast among others; every refusal with its message (sets of groups, "(Intercept)", counts without names, asymmetric covariances, negative contrast variances, a fit with approximate standard errors, the quantile's arguments); masked arrays (tests/test_masked_inputs.py); named contrasts matched by name, pandas included; the one-sided gene-level table; and the examples of the docstrings.

Each of 37 deliberate bugs ran in a copy of the package and made at least one test fail: 22 against the two test files as first written, 14 against the tests added in code review, and, after review restructured the code, seven of the first 22 again plus one more against tests/test_multivariate_t.py. At most three catching tests are listed:

Deliberate bug Tests that caught it
The two-sided tail of one coordinate counted once test_one_coordinate_is_the_t_distribution (3 cases)
The radial tail of two coordinates with exponent (df + 1) / 2 test_two_coordinates_match_mpmath, test_two_coordinate_quantiles_match_mpmath, test_two_coordinates_of_correlation_one_are_one_coordinate
The weight of the tilted chi draw with exponent (df + 1) / 2 test_product_correlations_match_dunnetts_integral_into_the_tail, test_rank_two_correlations_match_the_average_over_directions, test_stated_errors_are_honest
The conditioning threshold not scaled by S the same three
No steering of rare overlaps test_errors_stay_honest_deep_in_the_normal_tail
Coordinates of zero residual variance as indicators, not folded into a limit test_errors_stay_honest_with_nearly_collinear_coordinates, test_general_correlations_match_mvtnorm, test_rank_two_correlations_match_the_average_over_directions
Standard errors doubled test_stated_errors_are_honest
Standard errors halved test_equicorrelated_normal_matches_dunnetts_integral, test_product_correlations_match_dunnetts_integral_into_the_tail, test_stated_errors_are_honest
The guarded minimum of 16 points per shift lowered to 4 test_errors_stay_honest_with_nearly_collinear_coordinates
No guard for a coordinate correlated beyond 0.999 test_errors_stay_honest_with_nearly_collinear_coordinates
An absolute error target in place of the relative one test_multiple_contrast_test_matches_multcomp, test_product_correlations_match_dunnetts_integral_into_the_tail, test_small_boxes_keep_their_relative_accuracy
The complement side always, never the probability test_errors_stay_honest_with_nearly_collinear_coordinates, test_product_correlations_match_dunnetts_integral_into_the_tail, test_small_boxes_keep_their_relative_accuracy
The quadrature's tolerance relative to the larger side test_two_contrasts_need_no_random_numbers, test_two_coordinates_match_mpmath
No rounding floor in the quadrature's error test_two_coordinates_match_mpmath
The quantile's standard error halved test_product_correlation_quantiles_match_dunnetts_integral
Williams-type weights equal, not by group size test_contrast_matrices_match_multcomp, test_williams_weights_by_hand
Dunnett's control always the first group test_contrast_matrices_match_multcomp, test_dunnett_contrasts_take_groups_and_a_control
The contrasts' correlation ignored test_multiple_contrast_test_matches_multcomp, test_the_examples_run
"less" tested as "greater" test_less_at_t_is_greater_at_minus_t, test_multiple_contrast_test_matches_multcomp
No running maximum of p-values within a gene none at first, as the tests' data gave ordered p-values without it; test_a_less_extreme_contrast_never_gets_a_smaller_p_value, added, catches it
No Benjamini-Hochberg adjustment over genes test_the_tables
z-tests in place of t-tests test_multiple_contrast_test_matches_multcomp, test_one_contrast_is_the_moderated_t_test, test_p_values_never_decrease_as_t_becomes_less_extreme
A part's limits taken without the pivot order test_distinct_limits_match_dunnetts_integral (3 of 4 cases)
The probability side conditioned on coordinate 0 while its parts use the coordinate of smallest probability test_distinct_limits_match_dunnetts_integral (4 cases)
The quantile's standard error four times too large test_quantile_errors_are_honest_and_meet_the_target
The quantile stopping after its first round, whatever relative_error test_quantile_errors_are_honest_and_meet_the_target
Two coordinates left after infinite limits reading correlation[0, 1]; three reading the leading block test_infinite_limits_drop_their_coordinates_from_any_correlation (each)
The gene-level table choosing the contrast by |t| whatever the alternative test_the_overall_table_of_a_one_sided_test
A feature tested when any contrast, not every one, has positive variance test_a_contrast_of_zero_variance_makes_its_feature_missing
No check of negative contrast variances; none of symmetric covariances test_a_negative_variance_is_refused; test_asymmetric_covariances_are_refused
A moderated fit's common covariance for every gene; its covariance not scaled by the posterior variance test_a_moderated_fit_is_tested_with_its_own_covariances (weighted case; both cases)
The stored correlation left as the covariance test_the_correlation_of_dunnett_contrasts_and_simultaneous_intervals
Sets accepted as ordered groups test_contrast_builders_refuse (2 cases)
The lower and upper limits of the parts swapped when unpacked 27 tests, among them test_bounds_hold, test_a_coordinate_fixed_by_the_conditioning_one and test_distinct_limits_match_dunnetts_integral

Coverage of the final code, with NUMBA_DISABLE_JIT=1 (coverage 7.13.1, branches): cytepool/stats/multiple_contrasts.py 100% of 247 statements and 76 branches; cytepool/stats/multivariate_t.py 99% of 807 statements and 336 branches. Left: a guard that ends the two-coordinate quadrature when an interval cannot be split, which no input reached; the 200-step cap of the quantile's bisection, reached only by a root at exactly 0 with an error of exactly 0; and the two-sided limit of 0 of the box's slope, which the tests reach compiled. Uncompiled, t_sf_scalar of distributions raises at 0 (math.log(0)), which fails four tests that pass compiled, so the coverage run leaves them out, with the five that run thousands of uncompiled Monte Carlo problems.

Decisions

Decision Alternatives Reason
The complement as a sum of first exceedances, each conditioned on its exceedance, at every magnitude Genz's estimator of the inside probability (mvtnorm, scipy) with p = 1 - inside; the planned second-order Bonferroni bracket (Hunter 1976; Worsley 1982) with importance sampling and a stored bracket width In the prototype the inside estimator's error was near 100% below p = 1e-3, and it understated its own error by up to four orders of magnitude; the first-exceedance estimator gave about 1e-3 relative error at 512 points from p = 0.3 to 1e-10, with honest errors. One method needs no thresholds between methods, and the bracket would stay wide where the t's tail dependence matters
One contract, a standard error of at most relative_error (1%) of the smaller of the probability and the complement The planned absolute error of 1e-3, refined to 1e-5 near a decision threshold, and relative 1e-3 below p = 1e-3 Those regimes jump a thousandfold at p = 1e-3, and refining near a threshold couples the distribution function to decisions taken later across genes; 1e-3 everywhere costs about ten times more. At 1%, a call changes with the seed only for p-values within about 2% of the cut-off, and the error is stored with every p-value
The error relative to the smaller of the probability and the complement, for p-values too Relative to the p-value alone, tried during the build Small boxes need relative accuracy of the probability. Relative to p alone, p-values near 1 stopped after few points and understated their errors (a root-mean-square z of up to 21 over 100 seeds, against 1.00 to 1.24), though it cut the time per gene by 25% (Williams-type) and 34% (Dunnett's)
A deterministic quadrature for two coordinates, by the average over directions Ding's conditional distribution; the Monte Carlo path Seed-free p-values for the most common design in the survey, a vehicle and two doses, at 23 to 31 us per problem to 1e-12 relative, against 0.07 to 1.5 ms per term with Ding's formula, which became the independent reference
16 random shifts of extensible Kronecker points (square roots of primes) 8 or 10 shifts (Genz, scipy); scipy's lattice; Sobol' points A standard error on 15 degrees of freedom is reliable enough for the stopping rule and the calibrated tier; extensible points reuse the earlier ones when the points double; no tables; scipy stays an independent reference
At least 16 points per shift for parts with rare overlaps or a coordinate correlated beyond 0.999 with the one conditioned on 4 points per shift everywhere Both make features that 4 points per shift miss in every shift alike, so the spread of the shifts understates the error: measured root-mean-square z-scores of 2.1 to 3.4 before, 1.13 to 1.26 after
A new module, cytepool.stats.multivariate_t The planned cytepool.stats.distributions The distributions module is element-wise and never raises; these functions take matrices and random generators, validate and return errors, and 26 of 38 validation scripts reach distributions.py, whose change would mark them all as changed since their run
estimates and estimate_names, contrasts matched by name; builders that return Contrasts means, as first planned; contrasts read by position; plain matrices as contrMat returns The function takes any estimates and their covariance, as parm() does; CytePool never matches named weights by position; names travel with the weights into tables
A moderated fit in place of the estimates, giving their covariance, df and names A recipe that multiplies the posterior variance by the fit's unscaled covariance A recipe failed after contrast_standard_errors="limma" (no per-gene covariance), and its fallback, the design's covariance without weights, gave standard errors about three times too large at the median, without an error; the fit path refuses such fits and cannot take an unscaled covariance
williams_contrasts takes a mapping of group name to size only Also a sequence of counts, as contrMat(n) takes Counts named the groups "1", "2", ..., which matched no estimates but estimate_names=["1", "2", ...]; dunnett_contrasts already took names
Groups named "(Intercept)" and unordered sets refused Contrasts built from them With an intercept, the doses' coefficients are already differences from the vehicle, and "dose - (Intercept)" tests nothing meaningful; a set's order, and so the control, changed with Python's hash seed
The correlation of each gene's contrasts kept on the result multcomp's confint; no intervals Simultaneous confidence intervals take three lines with multivariate_t_quantile, which agree with scipy's dunnett, and PR 8's MCTtest critical value needs the same correlation
The quantile bisects between its exact bounds in every round Narrowing around the previous root, with loops that widen the bracket when the root moves out The bounds always hold the root, so the widening loops (52 lines, which no test reached) and their fallback go; the quantiles agree to 2.4e-10, and the search takes up to 39% longer, off the per-gene path
Williams-type contrasts named by the doses they pool, and allowed with two groups multcomp's "C 1", "C 2"; its refusal below three groups Readable in a table; a cell type left with one dose still gets its test, Dunnett's single contrast
df without a default, not necessarily an integer multcomp's default of 0 (z-tests); integer df A missing df would silently give z-tests; moderated df are not integers
One overall p-value per gene, from its most extreme contrast, with Benjamini-Hochberg over genes; per-contrast tables without a cross-gene adjustment A separate global test; Benjamini-Hochberg over every contrast of every gene The max-t p-value is the gene's test, already computed; an adjustment of within-gene p-values over genes controls no named error rate, and PR 8 adds stage-wise control after the gene-level screen
Common random shifts for every problem of a call Fresh draws per problem Results do not depend on the order of the problems, on subsets or on threads

Trimmed, and where each went:

  • The three accuracy regimes and the second-order Bonferroni bracket with importance sampling and a stored bracket width: replaced by the first-exceedance estimator and one contract, above. The Bonferroni bounds remain as test invariants.
  • Tail p-values against mvtnorm's pmvt with abseps = 1e-12: pmvt cannot reach them, as the information rows show, so mpmath, Dunnett's integral and ALOE give the tail references, down to complements of 1e-12.
  • Inclusion-exclusion in three dimensions as a tail reference: replaced by the average over directions in mpmath, exact for every rank-two correlation, singular ones included, and by ALOE for higher ranks.
  • A function for simultaneous confidence intervals (multcomp's confint) and step-down adjustments (Shaffer, Westfall): out; PR 8's lowest effective dose uses its own closed test, and the chapter gives intervals in three lines from the stored correlation and multivariate_t_quantile.

Left for later: resolving p-values near 1 with fewer points while keeping their errors honest, which would bring the Williams-type test closer to Dunnett's cost; and dropping the x^2 map that steers draws of rare overlaps, which a broad check found neither helps nor harms (a pooled root-mean-square z of 1.146 both ways over 183 problems and 30 seeds each) but whose removal needs another validation run.

Dunnett's and Williams-type tests refer the largest of correlated t-statistics to the multivariate t distribution, which CytePool could not compute. mvtnorm's default absolute error of 1e-3 cannot resolve the p-values that calls across 20,000 genes need, whose Benjamini-Hochberg cut-off for the top gene is 2.5e-6.

multivariate_t_cdf computes the probability of a box and its complement, and multivariate_t_quantile the critical value of a max-t test, each with its standard error, at any df, integer or not. The complement is a sum of first exceedances, each conditioned on its exceedance and integrated with 16 randomly shifted lattices, so that the standard error stays at most relative_error (1% by default) of the smaller of the probability and the complement, down to the smallest doubles. Two coordinates are integrated over directions by adaptive quadrature, without random numbers; singular correlations, infinite limits and NaN are handled.
A dose series asks which doses differ from the vehicle in each gene. Bonferroni's correction ignores the correlation that the shared vehicle creates, and an F-test says whether any dose differs, not which.

multiple_contrast_test runs single-step multiple contrast tests over every gene from estimates and their covariances, as multcomp's glht does with parm(): estimates, standard errors, t-statistics and p-values adjusted for the gene's contrasts, with a table of each gene's max-t test and its Benjamini-Hochberg adjustment over genes. dunnett_contrasts and williams_contrasts build each dose against the vehicle, and the vehicle against the size-weighted means of the highest doses, as Contrasts matched to the estimates by name.
validation/multiple_contrast_tests.py compares the probabilities with Ding's conditional distribution and the average over directions in mpmath, Dunnett's integral, mvtnorm and scipy; the quantiles with their roots, with mvtnorm and by a round trip; and the contrast matrices and tests with multcomp, whose p-values it compares where its absolute error of 1e-3 resolves them. Its mpmath references divide each integrand by its largest value at the breakpoints first, as mpmath's quadrature stops at an absolute error of 1e-30 and left orthants of 1e-51 and 1e-87 with 9 correct digits.

The tests read the fixtures the script writes, besides closed forms, Dunnett's integral computed in the test and invariants of the tests, and check that the stated errors are honest over hundreds of seeds.
"Multiple testing" gains a section on several contrasts of one gene: the max-t test, single-step p-values against Bonferroni's, Dunnett's and Williams-type contrasts, how the probabilities are computed and why their error is relative. The bulk RNA-seq guide tests both drugs against the control at one error rate per gene and compares the genes it calls with the F-test's.

The Validation page gains the section of multiple_contrast_tests.py, its summary rows, its benchmark and its commands, and records its run; "Choosing a method", the glossary and the bibliography follow.
The data of the other tests give p-values in order without it, so removing the running maximum went unnoticed. The test feeds it a less extreme contrast whose estimate fell below a more extreme one's, and checks that it takes that p-value and its error.
…inate

Each round of the quantile search narrowed its bracket around the previous root
and needed two loops to widen it again when more points moved the root outside;
no test reached those loops, and five calls repeated sixteen arguments. Nothing
compared limits that differ between coordinates with a reference, so a part's
limits taken in the wrong order passed every test.

Each round now bisects again between the univariate quantile and Bonferroni's,
which always hold the root, and clamps the interpolated root to them; the slope
comes from the final bracket, and the parts and fixed arguments travel as tuples.
Quantiles agree with the previous search to 2.4e-10, at up to 39% more time off
the per-gene path. Unreachable branches for infinite limits are gone, and the
docstrings describe the steering of rare overlaps as it is.

New tests compare limits per coordinate, tails and small boxes, with Dunnett's
integral; check that quantile errors are honest over 100 seeds and meet their
target; drop infinite limits from a general correlation; and cover a coordinate
fixed by the conditioning one, draws that underflow and rays parallel to a face.
Each catches a deliberate bug that the earlier tests missed.
…at mislead

Testing a fit's contrasts meant multiplying its posterior variance by an unscaled
covariance from its internals. After limma's approximate standard errors there was
no such covariance, and the fallback, the design's covariance without weights, gave
standard errors about three times too large without an error. Counts as Williams
sizes named groups "1", "2", ..., a set of groups gave a control by hash order,
"(Intercept)" made contrasts against an intercept, and asymmetric or negative
covariances gave messages about arguments never passed, or NaN rows.

multiple_contrast_test takes a ModeratedFit in place of the estimates, with its
covariance, df_total and names, and refuses approximate standard errors. The
builders refuse sets and "(Intercept)"; williams_contrasts takes named sizes only.
Covariances must be symmetric, a negative contrast variance names its feature, and
contrast_matrix words its messages for the estimates through target and remedy,
which replaces a second copy of its name matching. The result keeps each feature's
correlation of contrasts for simultaneous intervals, and the gene-level table
carries the chosen contrast's estimate and standard error.

Tests check the fit path against contrasts_fit, the correlation by hand and the
intervals against scipy's dunnett, the one-sided gene-level table, a zero-variance
contrast among others, and each refusal; each catches a deliberate bug.
…n level

The chapter said Bonferroni's p-values were 20% larger than the single-step ones
at 5 df, which is how much smaller the single-step ones are, and claimed that 2 to
4 replicates give small moderated df. The guide named a drug wherever its p-value
was below 0.05, which keeps no error rate across the genes called, and the chapter
read the largest Williams-type t as the number of doses that changed.

The chapter gives Bonferroni's excess by Dunnett's integral at 5 to 40 df and for
z-tests, points to df_total, runs its example on a simulated dose series with a
moderated fit, confirms contrasts at alpha R / G after the screen of the genes, as
stageR does (Van den Berge et al. 2017), says what a Williams-type contrast means
and that genes, whose direction is unknown, take two-sided tests, and shows
simultaneous intervals from the stored correlation. The guide names drugs at 0.0025
and splits its false calls: 27 of 483 genes, 26 of them changed by neither drug.
Williams (1971) is cited, and the glossary keeps its order.
No reference checked limits that differ between coordinates, and none checked
p-values below 1e-3 for correlations without a product structure, such as
Williams-type contrasts of four or more doses, which are the headline one-sided
test and the input of the dose-response tests to come.

The script adds 96 boxes of product correlations with a limit per coordinate,
against Dunnett's integral and in both calibration pools, and 54 complements from
1e-4 to 1e-10 of Williams-type, Tukey's and random contrasts against ALOE (Owen,
Maximov and Chertkov 2019), plain Monte Carlo from the mixture of the distributions
given each exceedance, written from the paper in the reference process. Both take
their inputs from generators of their own, so the other rows keep theirs.
… work

With a moderated fit, contrasts that did not match its coefficients asked for
estimate_names, which the fit path refuses; and the refusal of "(Intercept)" among
the groups advised testing a fit with an intercept as it is, which puts the
intercept into each gene's family of contrasts.

A fit's mismatch now speaks of the fit's coefficients and points to their names,
and the intercept's refusal says to test the other coefficients without it, or to
fit a design without an intercept. The chapter says to name the doses' coefficients
as the fit's contrasts, as the bulk RNA-seq guide does.
The validation script gained rows of limits per coordinate and of ALOE, and the
code it checks changed in this round, so its section, its digest and the totals of
the Validation page were out of date.

The section opens with the second run of the script on the final code: 118 checks,
106 pass, none failed, with the new rows and calibration pools, while every other
row came out as the build's run printed it; the references are listed as bullets,
the reference programs' details moved into the script, and the benchmark of the
same container gives the timings. The digest of the code that run checked is
recorded, and the summary row says which correlations reach which tails.
The guide's saved outputs named a version built from a working tree with changes
not yet committed. They now come from the committed code, and print the same
results.
The quadrature of boxes of two coordinates uses the nodes and weights of
QUADPACK's 15-point Gauss-Kronrod rule, which only a code comment named, while
CytePool cites coefficients taken from books in the References section of the
module that uses them. The module now cites Piessens et al. (1983), checked
against Crossref, OpenAlex and Open Library, and the bibliography holds the book.

The validation script's description of its calibrated tier now names the standard
errors in its formula, which read as if the values themselves were combined.
The dose chapter says that each dose against the vehicle with control over a gene's several doses, as Dunnett's test gives it, is not implemented. multiple_contrast_test now gives it: the moderated fit of a cell type from dose_response holds the covariance of its dose contrasts, and the single-step test of those contrasts is Dunnett's test.

The chapter now tells readers to pass the cell type's moderated fit and its dose contrasts to multiple_contrast_test, with a link to the section that explains the test.
The bulk RNA-seq guide printed the table of the multiple contrast test with 60 floats at full precision. CI checks a guide's saved outputs by comparing printed text exactly, and on GitHub's runners with AVX-512, numpy computes float64 logarithms, powers and exponentials with other last bits than on an AVX2 machine, so those digits can differ there, while they tell the reader nothing.

The cell now prints the five genes of smallest p-value, with the contrast, the estimate to 2 decimals and the p-value and adjusted p-value to 2 significant digits, then the error of the p-values as a share of each p-value, at the median and at most. The paragraph after it names the columns that the table holds and the cell leaves out.

Each printed value lies at least 2e-4 (relative) from a rounding boundary, and the p-values of the top six genes lie at least 6.7% apart, so differences in the last bits cannot change the text, and the reader sees the digits that carry information.
The dose response guide showed the first five rows of the hepatocytes' spline table as a raw table, with 20 F-statistics and p-values at full precision. CI checks a guide's saved outputs by comparing printed text exactly, and on GitHub's runners with AVX-512, numpy computes float64 logarithms, powers and exponentials with other last bits than on an AVX2 machine, so those digits can differ there, while they tell the reader nothing.

The cell now prints the same five genes under the table's column names, which the paragraph after it explains, with the F-statistic to 1 decimal and the p-value and both adjusted p-values to 2 significant digits.

Each printed value lies at least 5e-5 (relative) from a rounding boundary, and the p-values of the top six genes lie at least 0.46% apart, so differences in the last bits cannot change the text.
@parashardhapola
parashardhapola merged commit 2c102f2 into master Nov 20, 2026
6 checks passed
@parashardhapola
parashardhapola deleted the dose/04-multivariate-t-and-contrast-tests branch November 20, 2026 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant